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ABSTBACT ' 

^ * . The SOCRAIBS Cdmputer Assisted Test Retrieval Sy^tea 

is ;a bank of test, questions and answers for 11 subject matter areas. 
The sysfem has be^n available since 1974 to faculty and students of 
the^CaMfotnia State University and College jSystem. Both batch-mode 
aid 'in^^xlactive versions^ltre available. Telephone requests for test 
forms bx reproduction masters can be made one^ day and received by ^ 
coiirier ^he folloviiyg day. Users with data-links to the cent'ral 
processot may have tests pxinted locally on a high speed computer. 
Comjfuterized test retrieval and test scoring should be an important 
resource tO/ busy teachers, but fkctSlty have been slow "to accept 
computer assisted test construction. - (CH) 



-- ■ ' 

IT^ Documents acquired by ERIC include m^any informal unpublished ♦ 

♦^materials not available from other sources. ERIC nfakes every effort ♦ 

♦ to obtain the best copy available. Nevertheless, items of marginal ♦ 

♦ reproducibility are often encountered and this a^ffects the quality ♦ 

♦ bf thQ microfiche and hardcopy reproductions EBaC makes availably ♦ 

♦ via the ERXC Document Reproduction Service (EDBS) . EDRS is not, ♦ 

♦ responsible/for the quality of the original document.- Reproductions ♦ 

♦ supplied by EDRS are the best thajp can be ma^e ffom th6 original. ♦ 



t 



1^ 



it 135 U.S.OErAHTMENTOlfHEAtTM, 

1 ' *■ EDUCATION *WtLF AW > 

■» ^ , • ^ NATIONAL INSTITUTE OF 

- ■ ^ ^. \ • ' . EDUCATION 

. M— .-^-i ' - THIS DOCUMENT HAS OEEW REPRO- 

, ^ SOCRATES^ TEST RETRIEVAL ouceo exactly as received from 

^ i ■ l-l.-l ..ii.i.»*i-f*ei-PW aiin ^rtl I B«BC THE PERSON DRORGANIJ^ATIDN ORIGIN- 

f AT THE CALIFORNIA STATE UN VERSITY AND COLLEGES ating it points of vieJTor opinions 

Ml 1 nC Vf^VlrWni^i*^ w ■ — w ^ StATEO 00 NOT NECESSARILY REPRE 

^ ■ ' SEt^TOFFlCIAL NATIONAL INSTITUTE OF 

OllverW Jr.. California State college, Domlnguez Hills and eouc^.c^ pcs,t,c«,« pcucv 
Van Willis, California State University, Fullerton 

SOCRATES Computer Assisted Test Retrieval is a service available to faculty and Students of the 
flinetcw. campus sytem of the California State University and Colleges. SOCRATES provides users witK 
mSe versfon machine scorable exercises over our state-wide computer network f'om^banks m e evcj 
SSt'subject matter areas. The banks range in si?e from a few hundred items (Tests and M«J''" f™'"^^^^^^^^ . 
San fifteen thousand items (American History). Two versions of SOCRATES ^'^^'^'^^^^'-yj^^^^^^ 
loic version' that allows users to obtain tests on high-speed printers and (2) an mteracuvc vcrs.oi^wh. h J^^y 
Bfe'accessed from any terminal on our time sharing network. There are a number of retrj^eval options. A.uscr on 
2ny campus may request a test by telephone, have it printed either on standard forms or "'"^"^"""^^f^^^^^^ , 
reproducUon masteVs and receive the test via courier. Tests requested by telephone before 8:00 PjM- ^yP'^^^y - 
are deliveretfby courier the following day. User-s on campuses that customarilyxstablish data-links with the , 
eDC3300 central processor in Los Angeles may elect to have their tests printed at their local campus high- 
speed printer. ~" ' 

Teachers have long maintained files of question, they feel are suitable for student evaluation. Whether in ^ 
shoe-boxes as individual items» or -as copies of old tests, these collections have provided an important 
Jst uctS resource for teachers in high schools, community colleges, and other -Vi^^;,^^^^^^^^^^^^ 
classes many contact hours, and marginal student assistant help is the rule. In this sense, item banks are not 
new Moreover, the assembly of tests from item j,ools has been a method used , 
organizations. The Educational Testing Service Ras been producmg achievemen ests in ^rn"'"" 
mathematics routinely by this method since 196S.^ However, the availability of item ba^ks through computer 
Svor^s has just recently begun to emerge as a novel and powerful tool of the individual classroom teacher. 
Mo^L oni hundred cLputer ^ccessibl^^ item banks have been built at schools and colleges in he UnUed 
StateVand Canada. Although their use seems to be increasing, still only a tiny mmority of faculty are regular 
u e s: the great majority make either marginal use of them or no use at all. This situation may «.st because of 
oW habits' becausi of a lack of awareness about the existence of the banks or because "^anV faculty ^« ^ 
intimidated by the computer and believe that anytliiQg connected with it requires an operating knowledge of . 
ophisticated gadgetry. There are some who. if not openly hostile to any kind o retrieval 'VJ^^^' ^''"^ 
way to denigrate its attributes. 6ne critic summed it up: "You are just automating what we all do. those of us , 
To have been active in the promotion of Computer Assisted Test Construction see he ac. of a t ^ " 
what we all do" as that of influencing a fundamental change in cducationa management. Moreover the nature 
of the interaction between computer and instructor is such that the use of item banks may very hk ly P^^^^^^^^ 
be one of the few applications of computers to education that is economical as well as Pcfa«°8>cally effective^ 
Once an item retrieval system has been put into operation it is but a small step to implement an automat c 
Korfng system which keeps records on the performance of the student p^ulation being served and monitor 
JhrcharacSrTstS of usage on each of the items. The teacher is then^presented a tool for instantaneous 
m n tor ng 0 studem achievement without the burdensome record-keeping procedure that causes so many 
Sers to avoid individualized instructional methods for large numbers of students. As a small part of the 
whole ar a ' computer information retrieval and manipulation. CATC combines the threat and promi c of 
wtablishing standards of achievement within an institution or a group of mstitutions and it demands the 
attention and criticisni of teachers anxious to move in new directions. » 



^ Serlcti've SOCRATES written by Mr. Dave Osterberg, Caiifomia State University, Chlco. 

'Epstein. Marion G., Educational Technology, XIII, 3 (1973). \ 
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The banks were assembled in a for^niSt compatible with the Classroom teachcrSjippoft System, .a 
retrieval system developed jointly by IBM and th&Los Angeles Unified School T^sijictrThis system'yai and 
still is in, use at our San Diego campus, but it is not compatible with the othemmpus computers. A forrtial 
institutional'commitment to the development of our item>anks b«gan tw& years ago when a small project was 
funded to collect and classify questions in the naturaf sciences. Three banks (chemistry, physics and biology), 
each containing more than two thousand items, were generat<fd during the project period. The following'year, 
funds were allocated for the development of the SOCRATES retrieval systp that would be compatible with 
, all campus computers (including time-sharing). This year a major commitment was made to develop two banks 
to be used for comprehensive examinations, one in general chemistry an(J/onc in accounting. The philosophjt 
behind our administrative support originates in the desire to elevate the stiident's responsibility in the learning \ 
process. The availability of large numbers of items in one subject matter area will allow a student to generate 
self-tests for a judgment on competency. It will allow teacljcrs to (jncourage more students to work at| their own 
pace, without the usual accompanying dpluge ofpajicrwork rtsulting from having to evaluate manually 
numerous students at various stages of progrejvTnc generation 6f items is completely under teacher control. 
No "knowledge of programming or computer use is required ti take maximum advantage of the system. 
SOCRATES retrieves items by subject categoiy, difficulty, behaVior level (requiring knowledge or applicatjoji 
of knowledge) and keyword. Questions which require the use 6f additional material (enhanced iteris) and 
questions which always occur toge^ther (macro items) are allowed. A test may contain up to one hundi'ed fifty 
items; each lest may be edited (items added or removed) up to ninety-nine times. The. printing of tests on 
continuous form ditto masters' is a standard feature and $ user may request up to ten scrambled versions of each 
test. TJie insiructor's-nime required to prepare a test is significantly redirected. Rather than being involved in 
the mechanics of producing and grading the test, his ^mc is, spent in selecting appropriate items using the 
request and edit feature of SOCRATES and in evaluating student performancey . . . 

"The subject matter content classification system is the key to the organiz'ition of each bajik. There isj 
always a problem of this classification system reflecting personal prejudices of its author. Although this^seems 

be a somewhat greater problem in fields other than the natural sciences and mathematics, all authorso 
classifiqation systems have been encouraged to seek opinions on its organization from theirco^lcagucs-atm to 
avoid an>SRCcific teaching approaches, such as programmed instruction, sclf-paccdlcatningTran emphasis on 
behavioral o^c^ives, in order to gain the greater user acceptance. 



\ The Chemistry Bahk Project: A Case Study 



Collections of itcmsVthc discipline of freshman general chemistry were obtained (torn the Educational 
Testing Service and the MoW)uth, Oregon City Schools. Some additional items were oblain^from interested 
facuhiTmembcrs at several cJimpuses. These were coded for category and other selectioXparametcrs, 
ke^^lchcd and loaded, resulting in an initial bank of nearly four thousand individual questions. Th(^following 
manpower was required to build the bank: Vi man-year classifying, proof-reading and editing; '/4 man-year in 
the coordination of the e'fforts of the keypunch operators, the data processing service and those faculty doing 
the classifying; Vi man-ycar of keypunch -time. Since that time we have received additional items from bank 
coordinators at Eastern Michigan University, the University of Pittsburgh and Michigan State University. 
^XApproximalely two-thirds of these have been incorporated into our bank, increasing the size to more than ten 
ifibusand items. The bank is now being used for the sami evaluative ptoccdures as have been carried out 
manually for generations: quizzes, homework assignments, diagnostic tests, challenge examinations, term 
examinations and comprehensive examinations. Microfiche copies and hard copies of the contents of the bank 
are made available to students and faculty alike. The'implemenlalion of bank use has allowed the exchange oi 
testing methods and educational philosophies between the participating schools. Two examples are worth 
mentioning. A chemistry department at one of our camPJises regularly administers a diagnostic test to 
incoming students to aid in placing them in the appropriate chemistry cOurse. The lest was placed in the 
chemistry bank so that administration of it became routine. Colleagues on two other campuses learned of its 
availability and have now begun to use U.The student populations are quite differ-ent, as the three schools art 
located in urban, suburban and 'rural areas. A p/ojecl has been initiated to examine the corj-clalion, of the 

'Lippcy. G.. Togg?nburgcr. F.. Brown. C. D., AsStfciation for Educational Data Systems Journal. P. 75, March 1971. 
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diagnostic test.a«iorcs at each campus with other measures of student performance, such as SAT scores and high 
school grade point averages, in order to ei^luate the test as a general counselling tool. The second example is 
mor^ a general nature. Since the development of SOCRATES, numerous requests have been made for the, 
^hafige of chemistry item banks. There is obviously an active and dedicated ^roup of educators who are 
^MiM their own retrieval and testing systems. Since our costs showihat approximately $5-$ 10 are spent for 
each Solaced in the bank, it is to everyone's advantage to share in the production and development of these 
•items. We hSvc ehcouraged the exchange of banks (most frequently via magnetic tapes) at. every turn. 

' Thcnise of a shridard character set presented some initial difficulties because of the absence of numerous 
symbols used in chemicaljvbtation, most noticeably the lower case letters needed for certain elemental symbols, 
superscripts- for exponents and subscripts for chemical f;ormulas. The conventions that were adopted have 
proved to be serviceable if momcntariljuipsetting to one who is accustomed to the more- commonly accepted 
representations. Symbols for elements^e underlined4f they consist of two letters. Thus the symbol Hf is 
written HF to distinguish hafrtium from HF, hydrogen fluoride. Subscripts are shifted one Ijrtjidown so that 
calcium phosphate appears as CA,(POJ,-, and superscripts one line up; carbon-14 is written -JC^Multiplication 
is indicated with a single asterisk, A*B (the developers of the mathematics bank at San Diego State University 
found the nptation AXB to be. relatively unambiguous). Exponents are also shifted to the line above, C" 

rt'presentin^the square root of C. ChemiCal equations include the symbbls -.-,-.ss==-.===»-, and-. — , 

for.ctepiicai reactidh, chemical equilibrium and resonance, respectively. Greek letters have been spelled out, 
edefiped in the text or their numerical equivalents indicated. The Greek letter ir is written either as 3.14 or PI. 
* Electron dot formulas may be printed successfully by a judicious use of periods and colons. Graphical 
d&ignations must be performed either liy actually keypunching the pictures, which^s enorrtiously time- 
consuming or supplied separately in the figure file.for enhanced items. The items below illustrate some of these 
conventions: \ . • , 

t. THE VELOCITY OF A JBEAM" OP -NEUTRONS IN A NUCLEAR REACTOR IS 
2.0 #10 CMVSEC . VHAT IS THE DE BROGLIE WAVELENGTH OP 
TilESE NEUTRONS? (MASS OP NEUTRON, M = 1 .00699' AMU, 
i -27 

C H = 6.6 * 10.' ERG SEC (PLANCK' S^C^NST ANT) , LAMBDA = H/MY) 

^ -57 ^^<\2 - -10 

A. 5.4 * 10 CM B.-2.6 * 10 CM C^ 1.7 * 10 CM 

-8 / , -8 

D; 6.6 * 10 CM E. 2.0 »" 10 CM • 

2. THE EQUATION POR THE PREPARATION OP CALCIUM CHLORlDk IS 

„CACO + • 2HCL ' CACL ' + H 0 + CO , 

— 3 — 2 2 2 

100 73 111 ' 18 44-^ 

' WHERE THE NUMBERS BELOW THE FORMULAS REPRESENT THE RELATIVE 

I WEIGHT OP EACH COMPOUND. THE NUMBER OP GRAMS OP CACL THAT 

, ', ,2 

CAN BE FORMED FROM 80 GRAMS OF CACO IS ABOUT " 

O 

A. 89 B. 100 C. Ill D. 173 E. "NONEf OF THESE 
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Faculty acceptance of the bank. 



SOCRATES has been available to alV sixteen thousand faculty and nearly thre<^ hundred thousand 
students since its initial implementation in June, J974. The system is capable of delivering tests within minutes 
for a request placed from a local campus terminal gnd from several hours to two days for telephone requests 
(delivery by courier), depending upon the distance of the campus. Copies arc i)fcat and, in lime, Ihey will be 
cfror-free. Surely faculty will view this service as a path leading to further freedom for pducational innovation, 
particularly in the areas of modular education* self-testing and evon the traditional lecture-examination ap- 
proach^ Yet fagulty have been slow to accept Computer Assisted Test Construction/ It is clear that teachers 
have not yet developed the same partnership with the computer that they have enjoyed/ with their typewriters (or 
secretaries). Selecting items from a card file, assembling them and requesting a typed copy, or, more often than 
not, typing them oneself, is a method which holds a powerful grip on many teacher^. An added problem is that 
teachers perceive themselves to be a very discriminating group when it comes td selecting items. In our effort to 
be all things tp all users, we hav^read on the stylistic or stfbjective toes of some. A partial solution to this pro- 
blcrinTto develop large bar^ksln^^ of content, mode^ style and difficulty; secahd, it is necessary to update a 
bank frequently by a^drrignew items, deleting items with alarge user rejection /rate and editing errors. Since 
users assum^ths^^ test is tailor-made for them, they are piqued when, afteyone or two test modificaHons, 
the result i^jiSt^xcds^^^ and completely to their liking. On the other hand, students are a resourceful and har^ 
lot. ^aily of the caveats raised by instructors turn out to be relatively minor to the student when the time comc^j 
to read and interpret the' questions on the exam. In other words, the test is a useful measure of student.per- 
formance, depending less on exam content than on the preparation of the plass. Some instructors take this into 
consideration by generating two exums in the same subject area. One, including answers, is given to the student 
to study prior to the testing period. The second, with completely different (Questions but the same objectives is 
actually used to measure student achievement. 



0 are exposed to the capabilities of 
ow increase in its use, if for no other 
more students at various levels of 



We are encouraged by the great enthusiasm expressed by teachers 
Computer Assisted Test Construction and not at all discouraged by the 
reason than th^t it allows us to do what we are already doing, but U 
progress, less expen^ivelj;, more quickly and with fewer pfrors. It is a fafct that innovative products and techni- 
ques require years and sometimes decades to gain full acceptance. McLuhan's observation that "inj the age of 
Xerox, cycty nfan is a publisher*^ is even more appropriate for information retrieval from computers. We think 
that the awakening has occurred, but the giant is still groggy. 
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